{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e78e2bc2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/tjean/projects/hamilton/.venv/bin/python: No module named pip\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "# Execute this cell to install dependencies\n",
    "%pip install sf-hamilton[visualization]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52cb3087",
   "metadata": {},
   "source": [
    "# Hamilton notebook extension [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dagworks-inc/hamilton/blob/main/examples/jupyter_notebook_magic/example.ipynb) [![GitHub badge](https://img.shields.io/badge/github-view_source-2b3137?logo=github)](https://github.com/apache/hamilton/blob/main/examples/jupyter_notebook_magic/example.ipynb)\n",
    "\n",
    "Jupyter magics are commands that can be executed in notebooks using `%` and `%%` in code cells.\n",
    "- **Line magics** start with `%` and apply to the current line\n",
    "- **Cell magics** start with `%%`, need to be the first line of a cell, and apply to the entire cell.\n",
    "\n",
    " You can think of them as Python decorators for lines and cells.\n",
    "\n",
    "> For example, `%timeit complex_function()` will return the time to execute `complex_function()` and adding `%%timeit` will return the time to execute the entire cell.\n",
    "\n",
    "This notebook is a tutorial on the Hamilton Jupyter magics and how they can improve your interactive development experience. It is meant to be read and have all cells executed linearly.\n",
    "\n",
    "- **Section 2** - Dataflow definition\n",
    "- **Section 3** - Dataflow execution\n",
    "\n",
    "> ⚠ This notebook extension is something we're actively developing. If you find any bugs, edge cases, performance impacts, or if you have feature requests, let us know."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ad4517f",
   "metadata": {},
   "source": [
    "## 1. Loading the extension"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fb17a4b",
   "metadata": {},
   "source": [
    "To load our Jupyter Magic, we use `%load_ext` with the import path for the Python module (as if you did `import ...`). You only need to load it once, and will need to reload it if you restart the kernel just like you would for a Python module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e3b8a109",
   "metadata": {},
   "outputs": [],
   "source": [
    "# disable plugin autoloading for faster notebook start time\n",
    "from hamilton import registry\n",
    "registry.disable_autoload()\n",
    "\n",
    "%reload_ext hamilton.plugins.jupyter_magic\n",
    "from hamilton import driver  # we'll need this later"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "328d5e93",
   "metadata": {},
   "source": [
    "After loading the extension, Hamilton magics become available:\n",
    "- `%%cell_to_module`\n",
    "- `%module_to_cell`\n",
    "\n",
    "This notebook will cover them one-by-one, but if you need a quick refresher you can prepend `?` to get help."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2a3d64f3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001B[0;31mDocstring:\u001B[0m\n",
      "::\n",
      "\n",
      "  %cell_to_module [-m [MODULE_NAME]] [-d [DISPLAY]] [--display_cache]\n",
      "                      [-x [EXECUTE]] [-b BUILDER] [-c CONFIG] [-i INPUTS]\n",
      "                      [-o OVERRIDES] [--show_results] [-w [WRITE_TO_FILE]]\n",
      "                      [name]\n",
      "\n",
      "Turn a notebook cell into a Hamilton module definition. This allows you to define\n",
      "and execute a dataflow from a single cell.\n",
      "\n",
      "For example:\n",
      "```\n",
      "%%cell_to_module dataflow --display --execute\n",
      "def A() -> int:\n",
      "  return 37\n",
      "\n",
      "def B(A: int) -> bool:\n",
      "  return (A % 3) > 2\n",
      "```\n",
      "\n",
      "positional arguments:\n",
      "  name                  Name for the module defined in this cell.\n",
      "\n",
      "options:\n",
      "  -m <[MODULE_NAME]>, --module_name <[MODULE_NAME]>\n",
      "                        Alias for positional argument `module_name`. There for\n",
      "                        backwards compatibility. Prefer the position arg.\n",
      "  -d <[DISPLAY]>, --display <[DISPLAY]>\n",
      "                        Display the dataflow. The argument is the variable\n",
      "                        name of a dictionary of visualization kwargs; else {}.\n",
      "  --display_cache       After execution, display the retrieved results. This\n",
      "                        uses `dr.cache.view_run()`.\n",
      "  -x <[EXECUTE]>, --execute <[EXECUTE]>\n",
      "                        Execute the dataflow. The argument is the variable\n",
      "                        name of a list of nodes; else execute all nodes.\n",
      "  -b BUILDER, --builder BUILDER\n",
      "                        Builder to which the module will be added and used for\n",
      "                        execution. Allows to pass Config and Adapters\n",
      "  -c CONFIG, --config CONFIG\n",
      "                        Config to build a Driver. Passing -c/--config at the\n",
      "                        same time as a Builder -b/--builder with a config will\n",
      "                        raise an exception.\n",
      "  -i INPUTS, --inputs INPUTS\n",
      "                        Execution inputs. The argument is the variable name of\n",
      "                        a dict of inputs; else {}.\n",
      "  -o OVERRIDES, --overrides OVERRIDES\n",
      "                        Execution overrides. The argument is the variable name\n",
      "                        of a dict of overrides; else {}.\n",
      "  --show_results        Print node values in the output cell after each node\n",
      "                        is executed.\n",
      "  -w <[WRITE_TO_FILE]>, --write_to_file <[WRITE_TO_FILE]>\n",
      "                        Write cell content to a file. The argument is the file\n",
      "                        path; else write to {module_name}.py\n",
      "\u001B[0;31mFile:\u001B[0m      ~/projects/hamilton/hamilton/plugins/jupyter_magic.py"
     ]
    }
   ],
   "source": [
    "?%%cell_to_module"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a128cf8a",
   "metadata": {},
   "source": [
    "## 2. Define a Hamilton dataflow"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b19d4ea2",
   "metadata": {},
   "source": [
    "### 2.1 Basics\n",
    "The main magic is `%%cell_to_module MODULE_NAME` which turns a cell into a temporary Python module in-memory. Successful cell execution means it's a valid Hamilton dataflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ef73893f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"257pt\" height=\"212pt\"\n",
       " viewBox=\"0.00 0.00 256.50 212.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 208)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-208 252.5,-208 252.5,4 -4,4\"/>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>cluster__legend</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"8,-64 8,-196 104,-196 104,-64 8,-64\"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-180.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Legend</text>\n",
       "</g>\n",
       "<!-- joke_prompt -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>joke_prompt</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M236.5,-64C236.5,-64 141.5,-64 141.5,-64 135.5,-64 129.5,-58 129.5,-52 129.5,-52 129.5,-12 129.5,-12 129.5,-6 135.5,0 141.5,0 141.5,0 236.5,0 236.5,0 242.5,0 248.5,-6 248.5,-12 248.5,-12 248.5,-52 248.5,-52 248.5,-58 242.5,-64 236.5,-64\"/>\n",
       "<text text-anchor=\"start\" x=\"140.5\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">joke_prompt</text>\n",
       "<text text-anchor=\"start\" x=\"179.5\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- _joke_prompt_inputs -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>_joke_prompt_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"100.5,-54.5 11.5,-54.5 11.5,-9.5 100.5,-9.5 100.5,-54.5\"/>\n",
       "<text text-anchor=\"start\" x=\"27\" y=\"-27.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">topic</text>\n",
       "<text text-anchor=\"start\" x=\"67\" y=\"-27.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- _joke_prompt_inputs&#45;&gt;joke_prompt -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>_joke_prompt_inputs&#45;&gt;joke_prompt</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M100.62,-32C106.68,-32 113.02,-32 119.39,-32\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"119.41,-35.5 129.41,-32 119.41,-28.5 119.41,-35.5\"/>\n",
       "</g>\n",
       "<!-- input -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>input</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"85.5,-164.5 26.5,-164.5 26.5,-127.5 85.5,-127.5 85.5,-164.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-142.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">input</text>\n",
       "</g>\n",
       "<!-- function -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>function</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M84,-109.5C84,-109.5 28,-109.5 28,-109.5 22,-109.5 16,-103.5 16,-97.5 16,-97.5 16,-84.5 16,-84.5 16,-78.5 22,-72.5 28,-72.5 28,-72.5 84,-72.5 84,-72.5 90,-72.5 96,-78.5 96,-84.5 96,-84.5 96,-97.5 96,-97.5 96,-103.5 90,-109.5 84,-109.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-87.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">function</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7f471f3f8650>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%cell_to_module joke -d\n",
    "def joke_prompt(topic: str) -> str:\n",
    "    return f\"Tell me a short joke about {topic}\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47177b4f",
   "metadata": {},
   "source": [
    "The module name allows to namespace functions "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4b20538d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tell me a short joke about hello\n",
      "Tell me a short joke about greetings\n"
     ]
    }
   ],
   "source": [
    "print(joke.joke_prompt(topic=\"hello\"))\n",
    "print(joke_prompt(topic=\"greetings\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afa2d96c",
   "metadata": {},
   "source": [
    "### 2.2 Module imports\n",
    "Code found in cells with `%%cell_to_module` is treated like an isolated `.py` file. This means you need to define Python imports in the cell itself. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1117471d",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%cell_to_module joke\n",
    "from typing import Optional  # remove to get `NameError: name 'Optional' is not defined``\n",
    "\n",
    "def joke_prompt(topic: Optional[str] = None) -> str:\n",
    "    return f\"Tell me a short joke about {topic}\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bd36d80",
   "metadata": {},
   "source": [
    "### 2.3 Display module\n",
    "You can visualize with the `--display / -d` argument. It can receive a dictionary of [visualization `kwargs`](https://hamilton.apache.org/reference/drivers/Driver/#hamilton.driver.Driver.display_all_functions)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "19a7a66e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"257pt\" height=\"212pt\"\n",
       " viewBox=\"0.00 0.00 256.50 212.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 208)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-208 252.5,-208 252.5,4 -4,4\"/>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>cluster__legend</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"8,-64 8,-196 104,-196 104,-64 8,-64\"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-180.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Legend</text>\n",
       "</g>\n",
       "<!-- joke_prompt -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>joke_prompt</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M236.5,-64C236.5,-64 141.5,-64 141.5,-64 135.5,-64 129.5,-58 129.5,-52 129.5,-52 129.5,-12 129.5,-12 129.5,-6 135.5,0 141.5,0 141.5,0 236.5,0 236.5,0 242.5,0 248.5,-6 248.5,-12 248.5,-12 248.5,-52 248.5,-52 248.5,-58 242.5,-64 236.5,-64\"/>\n",
       "<text text-anchor=\"start\" x=\"140.5\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">joke_prompt</text>\n",
       "<text text-anchor=\"start\" x=\"179.5\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- _joke_prompt_inputs -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>_joke_prompt_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"100.5,-54.5 11.5,-54.5 11.5,-9.5 100.5,-9.5 100.5,-54.5\"/>\n",
       "<text text-anchor=\"start\" x=\"27\" y=\"-27.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">topic</text>\n",
       "<text text-anchor=\"start\" x=\"67\" y=\"-27.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- _joke_prompt_inputs&#45;&gt;joke_prompt -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>_joke_prompt_inputs&#45;&gt;joke_prompt</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M100.62,-32C106.68,-32 113.02,-32 119.39,-32\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"119.41,-35.5 129.41,-32 119.41,-28.5 119.41,-35.5\"/>\n",
       "</g>\n",
       "<!-- input -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>input</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"85.5,-164.5 26.5,-164.5 26.5,-127.5 85.5,-127.5 85.5,-164.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-142.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">input</text>\n",
       "</g>\n",
       "<!-- function -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>function</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M84,-109.5C84,-109.5 28,-109.5 28,-109.5 22,-109.5 16,-103.5 16,-97.5 16,-97.5 16,-84.5 16,-84.5 16,-78.5 22,-72.5 28,-72.5 28,-72.5 84,-72.5 84,-72.5 90,-72.5 96,-78.5 96,-84.5 96,-84.5 96,-97.5 96,-97.5 96,-103.5 90,-109.5 84,-109.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-87.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">function</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7f471e2682d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%cell_to_module joke --display\n",
    "def joke_prompt(topic: str) -> str:\n",
    "    return f\"Tell me a short joke about {topic}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4efeda39",
   "metadata": {},
   "outputs": [],
   "source": [
    "display_config = dict(orient=\"TB\")  # orient visualization top to bottom"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0fc841a4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"305pt\" height=\"181pt\"\n",
       " viewBox=\"0.00 0.00 304.50 181.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 177)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-177 300.5,-177 300.5,4 -4,4\"/>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>cluster__legend</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"114.5,-89 114.5,-165 288.5,-165 288.5,-89 114.5,-89\"/>\n",
       "<text text-anchor=\"middle\" x=\"201.5\" y=\"-149.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Legend</text>\n",
       "</g>\n",
       "<!-- joke_prompt -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>joke_prompt</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M107,-64C107,-64 12,-64 12,-64 6,-64 0,-58 0,-52 0,-52 0,-12 0,-12 0,-6 6,0 12,0 12,0 107,0 107,0 113,0 119,-6 119,-12 119,-12 119,-52 119,-52 119,-58 113,-64 107,-64\"/>\n",
       "<text text-anchor=\"start\" x=\"11\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">joke_prompt</text>\n",
       "<text text-anchor=\"start\" x=\"50\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- _joke_prompt_inputs -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>_joke_prompt_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"104,-138 15,-138 15,-93 104,-93 104,-138\"/>\n",
       "<text text-anchor=\"start\" x=\"30.5\" y=\"-111.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">topic</text>\n",
       "<text text-anchor=\"start\" x=\"70.5\" y=\"-111.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- _joke_prompt_inputs&#45;&gt;joke_prompt -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>_joke_prompt_inputs&#45;&gt;joke_prompt</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M59.5,-92.66C59.5,-87.04 59.5,-80.81 59.5,-74.55\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"63,-74.09 59.5,-64.09 56,-74.09 63,-74.09\"/>\n",
       "</g>\n",
       "<!-- input -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>input</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"280,-134 221,-134 221,-97 280,-97 280,-134\"/>\n",
       "<text text-anchor=\"middle\" x=\"250.5\" y=\"-111.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">input</text>\n",
       "</g>\n",
       "<!-- function -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>function</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M190.5,-134C190.5,-134 134.5,-134 134.5,-134 128.5,-134 122.5,-128 122.5,-122 122.5,-122 122.5,-109 122.5,-109 122.5,-103 128.5,-97 134.5,-97 134.5,-97 190.5,-97 190.5,-97 196.5,-97 202.5,-103 202.5,-109 202.5,-109 202.5,-122 202.5,-122 202.5,-128 196.5,-134 190.5,-134\"/>\n",
       "<text text-anchor=\"middle\" x=\"162.5\" y=\"-111.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">function</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7f471e25bb90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%cell_to_module joke --display display_config\n",
    "def joke_prompt(topic: str) -> str:\n",
    "    return f\"Tell me a short joke about {topic}\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51fbda38",
   "metadata": {},
   "source": [
    "### 2.4 Write module to file\n",
    "To make the transition from notebook to module easy and avoid copy-pasting, you can use `--write_to_file / -w`. This will copy the content of the file to `{MODULE_NAME}.py`. You can also specify a destination file path explicitly.\n",
    "\n",
    "> ⛔ Be careful with overwriting files with this command. Use git to version your files.\n",
    "\n",
    "After the running the next cell, you should see `joke.py` generated in your directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "a6f6052a",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%cell_to_module joke --write_to_file\n",
    "def joke_prompt(topic: str) -> str:\n",
    "    return f\"Knock, knock. Who's there? {topic}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "91875c91",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Knock, knock. Who's there? Cowsays\""
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import joke\n",
    "joke.joke_prompt(\"Cowsays\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db412fdd",
   "metadata": {},
   "source": [
    "### 2.5 Configure the `Driver`\n",
    "When using the `@config` function modifiers, you might need to pass a configuration to properly build your dataflow. You can do this inline with the `-c/--config` argument. It supports 3 different format:\n",
    "\n",
    "1. **Variable name** `--config my_config` where `my_config` is a variable e.g., `my_config=dict(a=True, b=-1)`\n",
    "2. **Key-value** `--config a=True, b=-1` evaluates to `dict(a=\"True\", b=\"-1\")` since everything is interpreted as strings.\n",
    "3. **JSON** `--config '{\"a\": true, \"b\": -1}'` evaluates to `dict(a=True, b=-1)`. For valid JSON, you need double quotes `\"` inside and have single quotes `'` outside.\n",
    "\n",
    "Here are examples. Notice how the config is properly displayed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6f00cdb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"359pt\" height=\"325pt\"\n",
       " viewBox=\"0.00 0.00 359.00 325.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 321)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-321 355,-321 355,4 -4,4\"/>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>cluster__legend</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"8,-123 8,-309 104,-309 104,-123 8,-123\"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-293.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Legend</text>\n",
       "</g>\n",
       "<!-- knock_joke -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>knock_joke</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"101,-50 5,-50 5,0 107,0 107,-44 101,-50\"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"101,-50 101,-44 \"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"107,-44 101,-44 \"/>\n",
       "<text text-anchor=\"start\" x=\"13\" y=\"-35.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">knock_joke</text>\n",
       "<text text-anchor=\"start\" x=\"41.5\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">true</text>\n",
       "</g>\n",
       "<!-- joke_prompt -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>joke_prompt</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M339,-123C339,-123 148,-123 148,-123 142,-123 136,-117 136,-111 136,-111 136,-71 136,-71 136,-65 142,-59 148,-59 148,-59 339,-59 339,-59 345,-59 351,-65 351,-71 351,-71 351,-111 351,-111 351,-117 345,-123 339,-123\"/>\n",
       "<text text-anchor=\"start\" x=\"147\" y=\"-101.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">joke_prompt: knock_joke</text>\n",
       "<text text-anchor=\"start\" x=\"234\" y=\"-73.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- _joke_prompt_inputs -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>_joke_prompt_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"100.5,-113.5 11.5,-113.5 11.5,-68.5 100.5,-68.5 100.5,-113.5\"/>\n",
       "<text text-anchor=\"start\" x=\"27\" y=\"-86.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">topic</text>\n",
       "<text text-anchor=\"start\" x=\"67\" y=\"-86.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- _joke_prompt_inputs&#45;&gt;joke_prompt -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>_joke_prompt_inputs&#45;&gt;joke_prompt</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M100.96,-91C108.66,-91 116.99,-91 125.62,-91\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"125.86,-94.5 135.86,-91 125.86,-87.5 125.86,-94.5\"/>\n",
       "</g>\n",
       "<!-- config -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>config</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"79.5,-278 26.5,-278 26.5,-242 85.5,-242 85.5,-272 79.5,-278\"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"79.5,-278 79.5,-272 \"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"85.5,-272 79.5,-272 \"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-256.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">config</text>\n",
       "</g>\n",
       "<!-- input -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>input</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"85.5,-223.5 26.5,-223.5 26.5,-186.5 85.5,-186.5 85.5,-223.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-201.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">input</text>\n",
       "</g>\n",
       "<!-- function -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>function</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M84,-168.5C84,-168.5 28,-168.5 28,-168.5 22,-168.5 16,-162.5 16,-156.5 16,-156.5 16,-143.5 16,-143.5 16,-137.5 22,-131.5 28,-131.5 28,-131.5 84,-131.5 84,-131.5 90,-131.5 96,-137.5 96,-143.5 96,-143.5 96,-156.5 96,-156.5 96,-162.5 90,-168.5 84,-168.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-146.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">function</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7f471e268ed0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%cell_to_module joke --display --config '{\"knock_joke\": \"true\"}'\n",
    "from hamilton.function_modifiers import config\n",
    "\n",
    "@config.when(knock_joke=\"true\")\n",
    "def joke_prompt__knock(topic: str) -> str:\n",
    "    return f\"Knock, knock. Who's there? {topic}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "758b3fbe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"359pt\" height=\"325pt\"\n",
       " viewBox=\"0.00 0.00 359.00 325.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 321)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-321 355,-321 355,4 -4,4\"/>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>cluster__legend</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"8,-123 8,-309 104,-309 104,-123 8,-123\"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-293.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Legend</text>\n",
       "</g>\n",
       "<!-- knock_joke -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>knock_joke</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"101,-50 5,-50 5,0 107,0 107,-44 101,-50\"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"101,-50 101,-44 \"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"107,-44 101,-44 \"/>\n",
       "<text text-anchor=\"start\" x=\"13\" y=\"-35.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">knock_joke</text>\n",
       "<text text-anchor=\"start\" x=\"41.5\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">true</text>\n",
       "</g>\n",
       "<!-- joke_prompt -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>joke_prompt</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M339,-123C339,-123 148,-123 148,-123 142,-123 136,-117 136,-111 136,-111 136,-71 136,-71 136,-65 142,-59 148,-59 148,-59 339,-59 339,-59 345,-59 351,-65 351,-71 351,-71 351,-111 351,-111 351,-117 345,-123 339,-123\"/>\n",
       "<text text-anchor=\"start\" x=\"147\" y=\"-101.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">joke_prompt: knock_joke</text>\n",
       "<text text-anchor=\"start\" x=\"234\" y=\"-73.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- _joke_prompt_inputs -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>_joke_prompt_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"100.5,-113.5 11.5,-113.5 11.5,-68.5 100.5,-68.5 100.5,-113.5\"/>\n",
       "<text text-anchor=\"start\" x=\"27\" y=\"-86.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">topic</text>\n",
       "<text text-anchor=\"start\" x=\"67\" y=\"-86.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- _joke_prompt_inputs&#45;&gt;joke_prompt -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>_joke_prompt_inputs&#45;&gt;joke_prompt</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M100.96,-91C108.66,-91 116.99,-91 125.62,-91\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"125.86,-94.5 135.86,-91 125.86,-87.5 125.86,-94.5\"/>\n",
       "</g>\n",
       "<!-- config -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>config</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"79.5,-278 26.5,-278 26.5,-242 85.5,-242 85.5,-272 79.5,-278\"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"79.5,-278 79.5,-272 \"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"85.5,-272 79.5,-272 \"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-256.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">config</text>\n",
       "</g>\n",
       "<!-- input -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>input</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"85.5,-223.5 26.5,-223.5 26.5,-186.5 85.5,-186.5 85.5,-223.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-201.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">input</text>\n",
       "</g>\n",
       "<!-- function -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>function</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M84,-168.5C84,-168.5 28,-168.5 28,-168.5 22,-168.5 16,-162.5 16,-156.5 16,-156.5 16,-143.5 16,-143.5 16,-137.5 22,-131.5 28,-131.5 28,-131.5 84,-131.5 84,-131.5 90,-131.5 96,-137.5 96,-143.5 96,-143.5 96,-156.5 96,-156.5 96,-162.5 90,-168.5 84,-168.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-146.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">function</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7f471e268790>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%cell_to_module joke --display --config knock_joke=true\n",
    "from hamilton.function_modifiers import config\n",
    "\n",
    "@config.when(knock_joke=\"true\")\n",
    "def joke_prompt__knock(topic: str) -> str:\n",
    "    return f\"Knock, knock. Who's there? {topic}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "073225f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "my_config = dict(knock_joke=\"true\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "311f0bad",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"359pt\" height=\"325pt\"\n",
       " viewBox=\"0.00 0.00 359.00 325.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 321)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-321 355,-321 355,4 -4,4\"/>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>cluster__legend</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"8,-123 8,-309 104,-309 104,-123 8,-123\"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-293.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Legend</text>\n",
       "</g>\n",
       "<!-- knock_joke -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>knock_joke</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"101,-50 5,-50 5,0 107,0 107,-44 101,-50\"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"101,-50 101,-44 \"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"107,-44 101,-44 \"/>\n",
       "<text text-anchor=\"start\" x=\"13\" y=\"-35.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">knock_joke</text>\n",
       "<text text-anchor=\"start\" x=\"41.5\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">true</text>\n",
       "</g>\n",
       "<!-- joke_prompt -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>joke_prompt</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M339,-123C339,-123 148,-123 148,-123 142,-123 136,-117 136,-111 136,-111 136,-71 136,-71 136,-65 142,-59 148,-59 148,-59 339,-59 339,-59 345,-59 351,-65 351,-71 351,-71 351,-111 351,-111 351,-117 345,-123 339,-123\"/>\n",
       "<text text-anchor=\"start\" x=\"147\" y=\"-101.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">joke_prompt: knock_joke</text>\n",
       "<text text-anchor=\"start\" x=\"234\" y=\"-73.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- _joke_prompt_inputs -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>_joke_prompt_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"100.5,-113.5 11.5,-113.5 11.5,-68.5 100.5,-68.5 100.5,-113.5\"/>\n",
       "<text text-anchor=\"start\" x=\"27\" y=\"-86.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">topic</text>\n",
       "<text text-anchor=\"start\" x=\"67\" y=\"-86.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- _joke_prompt_inputs&#45;&gt;joke_prompt -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>_joke_prompt_inputs&#45;&gt;joke_prompt</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M100.96,-91C108.66,-91 116.99,-91 125.62,-91\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"125.86,-94.5 135.86,-91 125.86,-87.5 125.86,-94.5\"/>\n",
       "</g>\n",
       "<!-- config -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>config</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"79.5,-278 26.5,-278 26.5,-242 85.5,-242 85.5,-272 79.5,-278\"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"79.5,-278 79.5,-272 \"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"85.5,-272 79.5,-272 \"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-256.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">config</text>\n",
       "</g>\n",
       "<!-- input -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>input</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"85.5,-223.5 26.5,-223.5 26.5,-186.5 85.5,-186.5 85.5,-223.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-201.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">input</text>\n",
       "</g>\n",
       "<!-- function -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>function</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M84,-168.5C84,-168.5 28,-168.5 28,-168.5 22,-168.5 16,-162.5 16,-156.5 16,-156.5 16,-143.5 16,-143.5 16,-137.5 22,-131.5 28,-131.5 28,-131.5 84,-131.5 84,-131.5 90,-131.5 96,-137.5 96,-143.5 96,-143.5 96,-156.5 96,-156.5 96,-162.5 90,-168.5 84,-168.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-146.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">function</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7f471e26bfd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%cell_to_module joke --display --config my_config\n",
    "from hamilton.function_modifiers import config\n",
    "\n",
    "@config.when(knock_joke=\"true\")\n",
    "def joke_prompt__knock(topic: str) -> str:\n",
    "    return f\"Knock, knock. Who's there? {topic}\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd00c4ad",
   "metadata": {},
   "source": [
    "### 2.6 Build a `Driver`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e65720a1",
   "metadata": {},
   "source": [
    "The `Driver` definition can be required to properly build some Hamilton dataflow, in particular those using `.with_config() / @config` and `Parallelizable[]/Collect[]`. We can pass a `Builder` object using `-b/--builder`.\n",
    "\n",
    "Here are examples. Notice how the config from the `Builder` is properly displayed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "f8fc4d76",
   "metadata": {},
   "outputs": [],
   "source": [
    "my_builder = (\n",
    "    driver.Builder()\n",
    "    .enable_dynamic_execution(allow_experimental_mode=True)\n",
    "    .with_config({\"knock_joke\": \"true\"})\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "3b924480",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"612pt\" height=\"423pt\"\n",
       " viewBox=\"0.00 0.00 612.00 423.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 419)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-419 608,-419 608,4 -4,4\"/>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>cluster__legend</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"11.5,-150 11.5,-407 109.5,-407 109.5,-150 11.5,-150\"/>\n",
       "<text text-anchor=\"middle\" x=\"60.5\" y=\"-391.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Legend</text>\n",
       "</g>\n",
       "<!-- knock_joke -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>knock_joke</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"105.5,-50 9.5,-50 9.5,0 111.5,0 111.5,-44 105.5,-50\"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"105.5,-50 105.5,-44 \"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"111.5,-44 105.5,-44 \"/>\n",
       "<text text-anchor=\"start\" x=\"17.5\" y=\"-35.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">knock_joke</text>\n",
       "<text text-anchor=\"start\" x=\"46\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">true</text>\n",
       "</g>\n",
       "<!-- joke_prompt -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>joke_prompt</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M353,-136C353,-136 162,-136 162,-136 156,-136 150,-130 150,-124 150,-124 150,-84 150,-84 150,-78 156,-72 162,-72 162,-72 353,-72 353,-72 359,-72 365,-78 365,-84 365,-84 365,-124 365,-124 365,-130 359,-136 353,-136\"/>\n",
       "<text text-anchor=\"start\" x=\"161\" y=\"-114.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">joke_prompt: knock_joke</text>\n",
       "<text text-anchor=\"start\" x=\"248\" y=\"-86.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- joke_prompt_collection -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>joke_prompt_collection</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"#ea5556\" d=\"M588,-136C588,-136 410,-136 410,-136 404,-136 398,-130 398,-124 398,-124 398,-84 398,-84 398,-78 404,-72 410,-72 410,-72 588,-72 588,-72 594,-72 600,-78 600,-84 600,-84 600,-124 600,-124 600,-130 594,-136 588,-136\"/>\n",
       "<path fill=\"none\" stroke=\"#ea5556\" d=\"M592,-140C592,-140 406,-140 406,-140 400,-140 394,-134 394,-128 394,-128 394,-80 394,-80 394,-74 400,-68 406,-68 406,-68 592,-68 592,-68 598,-68 604,-74 604,-80 604,-80 604,-128 604,-128 604,-134 598,-140 592,-140\"/>\n",
       "<text text-anchor=\"start\" x=\"409\" y=\"-114.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">joke_prompt_collection</text>\n",
       "<text text-anchor=\"start\" x=\"488.5\" y=\"-86.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">list</text>\n",
       "</g>\n",
       "<!-- joke_prompt&#45;&gt;joke_prompt_collection -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>joke_prompt&#45;&gt;joke_prompt_collection</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M375.54,-104C378.21,-104 380.89,-104 383.56,-104\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"375.29,-104 365.29,-99.5 370.29,-104 365.29,-104 365.29,-104 365.29,-104 370.29,-104 365.29,-108.5 375.29,-104 375.29,-104\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"383.82,-107.5 393.82,-104 383.82,-100.5 383.82,-107.5\"/>\n",
       "</g>\n",
       "<!-- topic -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>topic</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"#56e39f\" d=\"M105,-136C105,-136 16,-136 16,-136 10,-136 4,-130 4,-124 4,-124 4,-84 4,-84 4,-78 10,-72 16,-72 16,-72 105,-72 105,-72 111,-72 117,-78 117,-84 117,-84 117,-124 117,-124 117,-130 111,-136 105,-136\"/>\n",
       "<path fill=\"none\" stroke=\"#56e39f\" d=\"M109,-140C109,-140 12,-140 12,-140 6,-140 0,-134 0,-128 0,-128 0,-80 0,-80 0,-74 6,-68 12,-68 12,-68 109,-68 109,-68 115,-68 121,-74 121,-80 121,-80 121,-128 121,-128 121,-134 115,-140 109,-140\"/>\n",
       "<text text-anchor=\"start\" x=\"40.5\" y=\"-114.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">topic</text>\n",
       "<text text-anchor=\"start\" x=\"15\" y=\"-86.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">Parallelizable</text>\n",
       "</g>\n",
       "<!-- topic&#45;&gt;joke_prompt -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>topic&#45;&gt;joke_prompt</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M121.07,-104C127.09,-104 133.35,-104 139.72,-104\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"139.81,-104 149.81,-108.5 144.81,-104 149.81,-104 149.81,-104 149.81,-104 144.81,-104 149.81,-99.5 139.81,-104 139.81,-104\"/>\n",
       "</g>\n",
       "<!-- config -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>config</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"84,-376 31,-376 31,-340 90,-340 90,-370 84,-376\"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"84,-376 84,-370 \"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"90,-370 84,-370 \"/>\n",
       "<text text-anchor=\"middle\" x=\"60.5\" y=\"-354.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">config</text>\n",
       "</g>\n",
       "<!-- function -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>function</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M88.5,-321.5C88.5,-321.5 32.5,-321.5 32.5,-321.5 26.5,-321.5 20.5,-315.5 20.5,-309.5 20.5,-309.5 20.5,-296.5 20.5,-296.5 20.5,-290.5 26.5,-284.5 32.5,-284.5 32.5,-284.5 88.5,-284.5 88.5,-284.5 94.5,-284.5 100.5,-290.5 100.5,-296.5 100.5,-296.5 100.5,-309.5 100.5,-309.5 100.5,-315.5 94.5,-321.5 88.5,-321.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"60.5\" y=\"-299.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">function</text>\n",
       "</g>\n",
       "<!-- expand -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>expand</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"#56e39f\" d=\"M85.5,-262.5C85.5,-262.5 35.5,-262.5 35.5,-262.5 29.5,-262.5 23.5,-256.5 23.5,-250.5 23.5,-250.5 23.5,-237.5 23.5,-237.5 23.5,-231.5 29.5,-225.5 35.5,-225.5 35.5,-225.5 85.5,-225.5 85.5,-225.5 91.5,-225.5 97.5,-231.5 97.5,-237.5 97.5,-237.5 97.5,-250.5 97.5,-250.5 97.5,-256.5 91.5,-262.5 85.5,-262.5\"/>\n",
       "<path fill=\"none\" stroke=\"#56e39f\" d=\"M89.5,-266.5C89.5,-266.5 31.5,-266.5 31.5,-266.5 25.5,-266.5 19.5,-260.5 19.5,-254.5 19.5,-254.5 19.5,-233.5 19.5,-233.5 19.5,-227.5 25.5,-221.5 31.5,-221.5 31.5,-221.5 89.5,-221.5 89.5,-221.5 95.5,-221.5 101.5,-227.5 101.5,-233.5 101.5,-233.5 101.5,-254.5 101.5,-254.5 101.5,-260.5 95.5,-266.5 89.5,-266.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"60.5\" y=\"-240.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">expand</text>\n",
       "</g>\n",
       "<!-- collect -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>collect</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"#ea5556\" d=\"M82,-199.5C82,-199.5 39,-199.5 39,-199.5 33,-199.5 27,-193.5 27,-187.5 27,-187.5 27,-174.5 27,-174.5 27,-168.5 33,-162.5 39,-162.5 39,-162.5 82,-162.5 82,-162.5 88,-162.5 94,-168.5 94,-174.5 94,-174.5 94,-187.5 94,-187.5 94,-193.5 88,-199.5 82,-199.5\"/>\n",
       "<path fill=\"none\" stroke=\"#ea5556\" d=\"M86,-203.5C86,-203.5 35,-203.5 35,-203.5 29,-203.5 23,-197.5 23,-191.5 23,-191.5 23,-170.5 23,-170.5 23,-164.5 29,-158.5 35,-158.5 35,-158.5 86,-158.5 86,-158.5 92,-158.5 98,-164.5 98,-170.5 98,-170.5 98,-191.5 98,-191.5 98,-197.5 92,-203.5 86,-203.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"60.5\" y=\"-177.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">collect</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7f471e26c350>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%cell_to_module joke --display -b my_builder\n",
    "from hamilton.htypes import Parallelizable, Collect\n",
    "from hamilton.function_modifiers import config\n",
    "\n",
    "def topic() -> Parallelizable[str]:\n",
    "    for t in [\"Tom\", \"Jerry\"]:\n",
    "        yield t\n",
    "\n",
    "@config.when(knock_joke=\"true\")\n",
    "def joke_prompt__knock(topic: str) -> str:\n",
    "    return f\"Knock, knock. Who's there? {topic}\"\n",
    "\n",
    "def joke_prompt_collection(joke_prompt: Collect[str]) -> list:\n",
    "    return list(joke_prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "743a5e1e",
   "metadata": {},
   "source": [
    "### 2.7 Load external modules\n",
    "While developing your dataflow with `%%cell_to_module`, you might want to load nodes from another Python module. To do, simply import it and add it to the `Driver` using `.with_modules()` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "630ff804",
   "metadata": {},
   "outputs": [],
   "source": [
    "my_builder = driver.Builder().with_modules(joke)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c111f5d5",
   "metadata": {},
   "source": [
    "The nodes `topic` and `joke_prompt` origin from `joke.py`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "66dccada",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"387pt\" height=\"505pt\"\n",
       " viewBox=\"0.00 0.00 387.00 505.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 501)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-501 383,-501 383,4 -4,4\"/>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>cluster__legend</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"21,-231 21,-489 119,-489 119,-231 21,-231\"/>\n",
       "<text text-anchor=\"middle\" x=\"70\" y=\"-473.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Legend</text>\n",
       "</g>\n",
       "<!-- topic -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>topic</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"#56e39f\" d=\"M114.5,-68C114.5,-68 25.5,-68 25.5,-68 19.5,-68 13.5,-62 13.5,-56 13.5,-56 13.5,-16 13.5,-16 13.5,-10 19.5,-4 25.5,-4 25.5,-4 114.5,-4 114.5,-4 120.5,-4 126.5,-10 126.5,-16 126.5,-16 126.5,-56 126.5,-56 126.5,-62 120.5,-68 114.5,-68\"/>\n",
       "<path fill=\"none\" stroke=\"#56e39f\" d=\"M118.5,-72C118.5,-72 21.5,-72 21.5,-72 15.5,-72 9.5,-66 9.5,-60 9.5,-60 9.5,-12 9.5,-12 9.5,-6 15.5,0 21.5,0 21.5,0 118.5,0 118.5,0 124.5,0 130.5,-6 130.5,-12 130.5,-12 130.5,-60 130.5,-60 130.5,-66 124.5,-72 118.5,-72\"/>\n",
       "<text text-anchor=\"start\" x=\"50\" y=\"-46.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">topic</text>\n",
       "<text text-anchor=\"start\" x=\"24.5\" y=\"-18.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">Parallelizable</text>\n",
       "</g>\n",
       "<!-- joke_response -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>joke_response</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M329,-145C329,-145 219,-145 219,-145 213,-145 207,-139 207,-133 207,-133 207,-93 207,-93 207,-87 213,-81 219,-81 219,-81 329,-81 329,-81 335,-81 341,-87 341,-93 341,-93 341,-133 341,-133 341,-139 335,-145 329,-145\"/>\n",
       "<text text-anchor=\"start\" x=\"218\" y=\"-123.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">joke_response</text>\n",
       "<text text-anchor=\"start\" x=\"264.5\" y=\"-95.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- joke_prompt_collection -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>joke_prompt_collection</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"#ea5556\" d=\"M363,-231C363,-231 185,-231 185,-231 179,-231 173,-225 173,-219 173,-219 173,-179 173,-179 173,-173 179,-167 185,-167 185,-167 363,-167 363,-167 369,-167 375,-173 375,-179 375,-179 375,-219 375,-219 375,-225 369,-231 363,-231\"/>\n",
       "<path fill=\"none\" stroke=\"#ea5556\" d=\"M367,-235C367,-235 181,-235 181,-235 175,-235 169,-229 169,-223 169,-223 169,-175 169,-175 169,-169 175,-163 181,-163 181,-163 367,-163 367,-163 373,-163 379,-169 379,-175 379,-175 379,-223 379,-223 379,-229 373,-235 367,-235\"/>\n",
       "<text text-anchor=\"start\" x=\"184\" y=\"-209.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">joke_prompt_collection</text>\n",
       "<text text-anchor=\"start\" x=\"263.5\" y=\"-181.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">list</text>\n",
       "</g>\n",
       "<!-- _joke_response_inputs -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>_joke_response_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"140,-135.5 0,-135.5 0,-90.5 140,-90.5 140,-135.5\"/>\n",
       "<text text-anchor=\"start\" x=\"15\" y=\"-108.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">joke_prompt</text>\n",
       "<text text-anchor=\"start\" x=\"106\" y=\"-108.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- _joke_response_inputs&#45;&gt;joke_response -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>_joke_response_inputs&#45;&gt;joke_response</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M140.23,-113C158.45,-113 178.19,-113 196.75,-113\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"196.92,-116.5 206.92,-113 196.92,-109.5 196.92,-116.5\"/>\n",
       "</g>\n",
       "<!-- _joke_prompt_collection_inputs -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>_joke_prompt_collection_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"140,-221.5 0,-221.5 0,-176.5 140,-176.5 140,-221.5\"/>\n",
       "<text text-anchor=\"start\" x=\"15\" y=\"-194.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">joke_prompt</text>\n",
       "<text text-anchor=\"start\" x=\"106\" y=\"-194.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- _joke_prompt_collection_inputs&#45;&gt;joke_prompt_collection -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>_joke_prompt_collection_inputs&#45;&gt;joke_prompt_collection</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M140.23,-199C146.16,-199 152.25,-199 158.41,-199\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"158.82,-202.5 168.82,-199 158.82,-195.5 158.82,-202.5\"/>\n",
       "</g>\n",
       "<!-- input -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>input</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"99.5,-457.5 40.5,-457.5 40.5,-420.5 99.5,-420.5 99.5,-457.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"70\" y=\"-435.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">input</text>\n",
       "</g>\n",
       "<!-- function -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>function</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M98,-402.5C98,-402.5 42,-402.5 42,-402.5 36,-402.5 30,-396.5 30,-390.5 30,-390.5 30,-377.5 30,-377.5 30,-371.5 36,-365.5 42,-365.5 42,-365.5 98,-365.5 98,-365.5 104,-365.5 110,-371.5 110,-377.5 110,-377.5 110,-390.5 110,-390.5 110,-396.5 104,-402.5 98,-402.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"70\" y=\"-380.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">function</text>\n",
       "</g>\n",
       "<!-- expand -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>expand</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"#56e39f\" d=\"M95,-343.5C95,-343.5 45,-343.5 45,-343.5 39,-343.5 33,-337.5 33,-331.5 33,-331.5 33,-318.5 33,-318.5 33,-312.5 39,-306.5 45,-306.5 45,-306.5 95,-306.5 95,-306.5 101,-306.5 107,-312.5 107,-318.5 107,-318.5 107,-331.5 107,-331.5 107,-337.5 101,-343.5 95,-343.5\"/>\n",
       "<path fill=\"none\" stroke=\"#56e39f\" d=\"M99,-347.5C99,-347.5 41,-347.5 41,-347.5 35,-347.5 29,-341.5 29,-335.5 29,-335.5 29,-314.5 29,-314.5 29,-308.5 35,-302.5 41,-302.5 41,-302.5 99,-302.5 99,-302.5 105,-302.5 111,-308.5 111,-314.5 111,-314.5 111,-335.5 111,-335.5 111,-341.5 105,-347.5 99,-347.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"70\" y=\"-321.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">expand</text>\n",
       "</g>\n",
       "<!-- collect -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>collect</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"#ea5556\" d=\"M91.5,-280.5C91.5,-280.5 48.5,-280.5 48.5,-280.5 42.5,-280.5 36.5,-274.5 36.5,-268.5 36.5,-268.5 36.5,-255.5 36.5,-255.5 36.5,-249.5 42.5,-243.5 48.5,-243.5 48.5,-243.5 91.5,-243.5 91.5,-243.5 97.5,-243.5 103.5,-249.5 103.5,-255.5 103.5,-255.5 103.5,-268.5 103.5,-268.5 103.5,-274.5 97.5,-280.5 91.5,-280.5\"/>\n",
       "<path fill=\"none\" stroke=\"#ea5556\" d=\"M95.5,-284.5C95.5,-284.5 44.5,-284.5 44.5,-284.5 38.5,-284.5 32.5,-278.5 32.5,-272.5 32.5,-272.5 32.5,-251.5 32.5,-251.5 32.5,-245.5 38.5,-239.5 44.5,-239.5 44.5,-239.5 95.5,-239.5 95.5,-239.5 101.5,-239.5 107.5,-245.5 107.5,-251.5 107.5,-251.5 107.5,-272.5 107.5,-272.5 107.5,-278.5 101.5,-284.5 95.5,-284.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"70\" y=\"-258.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">collect</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7f471e25a290>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%cell_to_module reply --display --builder my_builder\n",
    "def joke_response(joke_prompt: str) -> str:\n",
    "    return f\"{joke_prompt}\\n\\nCowsay who?\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03d8b0ed",
   "metadata": {},
   "source": [
    "### 2.8 Edit external modules\n",
    "It is also possible to load the content of a Python module into a notebook cell to be able to edit it interactively!\n",
    "\n",
    "This is essentially the reverse operation of `%%cell_to_module` hence why it's called `%module_to_cell`. This is a line magic (single `%`) and it reads the content of the line as a file path to a `.py` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "8bcd9ef3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<script>\n",
       "\n",
       "        if (document.getElementById('notebook-container')) {\n",
       "            //console.log('Jupyter Notebook');\n",
       "            allCells = document.getElementById('notebook-container').children;\n",
       "            selectionClass = /\\bselected\\b/;\n",
       "            jupyter = 'notebook';\n",
       "        }\n",
       "        else if (document.getElementsByClassName('jp-Notebook-cell').length){\n",
       "            //console.log('Jupyter Lab');\n",
       "            allCells = document.getElementsByClassName('jp-Notebook-cell');\n",
       "            selectionClass = /\\bjp-mod-selected\\b/;\n",
       "            jupyter = 'lab';\n",
       "        }\n",
       "        else {\n",
       "            console.log('Unknown Environment');\n",
       "        }\n",
       "\n",
       "        if (typeof allCells !== 'undefined') {\n",
       "            for (i = 0; i < allCells.length - 1; i++) {\n",
       "                if(selectionClass.test(allCells[i].getAttribute('class'))){\n",
       "                    allCells[i + 1].remove();\n",
       "\n",
       "                    // remove output indicators of current cell\n",
       "                    window.setTimeout(function(){\n",
       "                        if(jupyter === 'lab') {\n",
       "                            allCells[i].setAttribute('class', allCells[i].getAttribute('class') + ' jp-mod-noOutputs');\n",
       "                            allCells[i].getElementsByClassName('jp-OutputArea jp-Cell-outputArea')[0].innerHTML = '';\n",
       "                        } else if(jupyter === 'notebook'){\n",
       "                            allCells[i].getElementsByClassName('output')[0].innerHTML = '';\n",
       "                        }\n",
       "                    }, 20);\n",
       "\n",
       "                    break;\n",
       "                }\n",
       "            }\n",
       "        }\n",
       "        </script>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# execute this to generate a new cell\n",
    "%module_to_cell ./joke.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "873793f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%cell_to_module joke\n",
    "def joke_prompt(topic: str) -> str:\n",
    "    return f\"Knock, knock. Who's there? {topic}\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f9aec0a",
   "metadata": {},
   "source": [
    "If you executed the previous cell, a new code cell was created above with the content of `joke.py`. You can add `--write_to_file` to write the notebook cell back to the file."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ecf1bdf",
   "metadata": {},
   "source": [
    "## 3. Execute a dataflow\n",
    "One of the best part about notebooks is the ability to execute and immediately inspect results. They provide a \"read-eval-print\" loop (REPL) coding experience. With this extension, you can use a single notebook cell to define and execute your dataflow for a tight feedback loop.\n",
    "\n",
    "If you're familiar with Hamilton, you probably figured out that you can build a `Driver` from the dynamically defined modules (like the next cell). But we have better interactive options!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "f704fe3f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'joke_prompt': None}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dr = driver.Builder().with_modules(joke).build()\n",
    "results = dr.execute([\"joke_prompt\"], inputs=dict(topic=\"Cowsay\"))\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa13f109",
   "metadata": {},
   "source": [
    "### 3.1 Execute cell\n",
    "By adding `--execute / -x` to your module definition, the defined dataflow will be executed using `Driver.execute()` with all available nodes.\n",
    "\n",
    "The `--display` visualization should now include **output** nodes reflecting the executed nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "7eebf289",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"348pt\" height=\"304pt\"\n",
       " viewBox=\"0.00 0.00 348.00 304.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 300)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-300 344,-300 344,4 -4,4\"/>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>cluster__legend</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"13,-156 13,-288 109,-288 109,-156 13,-156\"/>\n",
       "<text text-anchor=\"middle\" x=\"61\" y=\"-272.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Legend</text>\n",
       "</g>\n",
       "<!-- joke_prompt -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>joke_prompt</title>\n",
       "<path fill=\"#ffc857\" stroke=\"black\" d=\"M108.5,-64C108.5,-64 13.5,-64 13.5,-64 7.5,-64 1.5,-58 1.5,-52 1.5,-52 1.5,-12 1.5,-12 1.5,-6 7.5,0 13.5,0 13.5,0 108.5,0 108.5,0 114.5,0 120.5,-6 120.5,-12 120.5,-12 120.5,-52 120.5,-52 120.5,-58 114.5,-64 108.5,-64\"/>\n",
       "<text text-anchor=\"start\" x=\"12.5\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">joke_prompt</text>\n",
       "<text text-anchor=\"start\" x=\"51.5\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- reply -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>reply</title>\n",
       "<path fill=\"#ffc857\" stroke=\"black\" d=\"M201,-64C201,-64 163,-64 163,-64 157,-64 151,-58 151,-52 151,-52 151,-12 151,-12 151,-6 157,0 163,0 163,0 201,0 201,0 207,0 213,-6 213,-12 213,-12 213,-52 213,-52 213,-58 207,-64 201,-64\"/>\n",
       "<text text-anchor=\"start\" x=\"162\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">reply</text>\n",
       "<text text-anchor=\"start\" x=\"172.5\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- joke_prompt&#45;&gt;reply -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>joke_prompt&#45;&gt;reply</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M120.8,-32C127.48,-32 134.17,-32 140.53,-32\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"140.88,-35.5 150.88,-32 140.88,-28.5 140.88,-35.5\"/>\n",
       "</g>\n",
       "<!-- punchline -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>punchline</title>\n",
       "<path fill=\"#ffc857\" stroke=\"black\" d=\"M328,-64C328,-64 254,-64 254,-64 248,-64 242,-58 242,-52 242,-52 242,-12 242,-12 242,-6 248,0 254,0 254,0 328,0 328,0 334,0 340,-6 340,-12 340,-12 340,-52 340,-52 340,-58 334,-64 328,-64\"/>\n",
       "<text text-anchor=\"start\" x=\"253\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">punchline</text>\n",
       "<text text-anchor=\"start\" x=\"281.5\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- reply&#45;&gt;punchline -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>reply&#45;&gt;punchline</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M213.09,-32C218.93,-32 225.24,-32 231.64,-32\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"231.71,-35.5 241.71,-32 231.71,-28.5 231.71,-35.5\"/>\n",
       "</g>\n",
       "<!-- a_dataframe -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>a_dataframe</title>\n",
       "<path fill=\"#ffc857\" stroke=\"black\" d=\"M110,-146C110,-146 12,-146 12,-146 6,-146 0,-140 0,-134 0,-134 0,-94 0,-94 0,-88 6,-82 12,-82 12,-82 110,-82 110,-82 116,-82 122,-88 122,-94 122,-94 122,-134 122,-134 122,-140 116,-146 110,-146\"/>\n",
       "<text text-anchor=\"start\" x=\"11\" y=\"-124.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">a_dataframe</text>\n",
       "<text text-anchor=\"start\" x=\"23\" y=\"-96.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">DataFrame</text>\n",
       "</g>\n",
       "<!-- function -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>function</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M89,-256.5C89,-256.5 33,-256.5 33,-256.5 27,-256.5 21,-250.5 21,-244.5 21,-244.5 21,-231.5 21,-231.5 21,-225.5 27,-219.5 33,-219.5 33,-219.5 89,-219.5 89,-219.5 95,-219.5 101,-225.5 101,-231.5 101,-231.5 101,-244.5 101,-244.5 101,-250.5 95,-256.5 89,-256.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"61\" y=\"-234.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">function</text>\n",
       "</g>\n",
       "<!-- output -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>output</title>\n",
       "<path fill=\"#ffc857\" stroke=\"black\" d=\"M83,-201.5C83,-201.5 39,-201.5 39,-201.5 33,-201.5 27,-195.5 27,-189.5 27,-189.5 27,-176.5 27,-176.5 27,-170.5 33,-164.5 39,-164.5 39,-164.5 83,-164.5 83,-164.5 89,-164.5 95,-170.5 95,-176.5 95,-176.5 95,-189.5 95,-189.5 95,-195.5 89,-201.5 83,-201.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"61\" y=\"-179.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">output</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7f471e26ef90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%cell_to_module joke --display --execute\n",
    "import pandas as pd\n",
    "\n",
    "def joke_prompt() -> str:\n",
    "    return f\"Knock, knock. Who's there? Cowsay\"\n",
    "\n",
    "def reply(joke_prompt: str) -> str:\n",
    "    _, _, right = joke_prompt.partition(\"? \")\n",
    "    return f\"{right} who?\"\n",
    "\n",
    "def punchline(reply: str) -> str:\n",
    "    left, _, _ = reply.partition(\" \")\n",
    "    return f\"No, {left} MooOOooo\"\n",
    "\n",
    "def a_dataframe() -> pd.DataFrame:\n",
    "    return pd.DataFrame({\"a\": [0, 1, 2, 3], \"b\": [\"a\", \"b\", \"c\", \"d\"]})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55b030df",
   "metadata": {},
   "source": [
    "Adding the flag `--show_results` will print node results to the cell output after each execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "ac13d20a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KeyError: Received `--inputs docs` but variable not found.\n"
     ]
    }
   ],
   "source": [
    "%%cell_to_module joke --execute --inputs docs\n",
    "import pandas as pd\n",
    "\n",
    "def joke_prompt() -> str:\n",
    "    return f\"Knock, knock. Who's there? Cowsay\"\n",
    "\n",
    "def reply(joke_prompt: str) -> str:\n",
    "    _, _, right = joke_prompt.partition(\"? \")\n",
    "    return f\"{right} who?\"\n",
    "\n",
    "def punchline(reply: str) -> str:\n",
    "    left, _, _ = reply.partition(\" \")\n",
    "    return f\"No, {left} MooOOooo\"\n",
    "\n",
    "def a_dataframe() -> pd.DataFrame:\n",
    "    return pd.DataFrame({\"a\": [0, 1, 2, 3], \"b\": [\"a\", \"b\", \"c\", \"d\"]})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68f55086",
   "metadata": {},
   "source": [
    "### 3.2 Requesting nodes\n",
    "You can a variable name to `--execute` which specifies the list of nodes to execute. This will be reflected in the visualization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "8721d077",
   "metadata": {},
   "outputs": [],
   "source": [
    "node_to_execute = [\"reply\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "a3e8da7f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"343pt\" height=\"222pt\"\n",
       " viewBox=\"0.00 0.00 343.00 222.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 218)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-218 339,-218 339,4 -4,4\"/>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>cluster__legend</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"8,-74 8,-206 104,-206 104,-74 8,-74\"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-190.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Legend</text>\n",
       "</g>\n",
       "<!-- joke_prompt -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>joke_prompt</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M232,-64C232,-64 137,-64 137,-64 131,-64 125,-58 125,-52 125,-52 125,-12 125,-12 125,-6 131,0 137,0 137,0 232,0 232,0 238,0 244,-6 244,-12 244,-12 244,-52 244,-52 244,-58 238,-64 232,-64\"/>\n",
       "<text text-anchor=\"start\" x=\"136\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">joke_prompt</text>\n",
       "<text text-anchor=\"start\" x=\"175\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- reply -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>reply</title>\n",
       "<path fill=\"#ffc857\" stroke=\"black\" d=\"M323,-64C323,-64 285,-64 285,-64 279,-64 273,-58 273,-52 273,-52 273,-12 273,-12 273,-6 279,0 285,0 285,0 323,0 323,0 329,0 335,-6 335,-12 335,-12 335,-52 335,-52 335,-58 329,-64 323,-64\"/>\n",
       "<text text-anchor=\"start\" x=\"284\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">reply</text>\n",
       "<text text-anchor=\"start\" x=\"294.5\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- joke_prompt&#45;&gt;reply -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>joke_prompt&#45;&gt;reply</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M244.25,-32C250.42,-32 256.59,-32 262.48,-32\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"262.74,-35.5 272.74,-32 262.74,-28.5 262.74,-35.5\"/>\n",
       "</g>\n",
       "<!-- topic -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>topic</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M75,-64C75,-64 37,-64 37,-64 31,-64 25,-58 25,-52 25,-52 25,-12 25,-12 25,-6 31,0 37,0 37,0 75,0 75,0 81,0 87,-6 87,-12 87,-12 87,-52 87,-52 87,-58 81,-64 75,-64\"/>\n",
       "<text text-anchor=\"start\" x=\"36\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">topic</text>\n",
       "<text text-anchor=\"start\" x=\"46.5\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- topic&#45;&gt;joke_prompt -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>topic&#45;&gt;joke_prompt</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M87.3,-32C95.71,-32 105.2,-32 114.85,-32\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"114.94,-35.5 124.94,-32 114.94,-28.5 114.94,-35.5\"/>\n",
       "</g>\n",
       "<!-- function -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>function</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M84,-174.5C84,-174.5 28,-174.5 28,-174.5 22,-174.5 16,-168.5 16,-162.5 16,-162.5 16,-149.5 16,-149.5 16,-143.5 22,-137.5 28,-137.5 28,-137.5 84,-137.5 84,-137.5 90,-137.5 96,-143.5 96,-149.5 96,-149.5 96,-162.5 96,-162.5 96,-168.5 90,-174.5 84,-174.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-152.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">function</text>\n",
       "</g>\n",
       "<!-- output -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>output</title>\n",
       "<path fill=\"#ffc857\" stroke=\"black\" d=\"M78,-119.5C78,-119.5 34,-119.5 34,-119.5 28,-119.5 22,-113.5 22,-107.5 22,-107.5 22,-94.5 22,-94.5 22,-88.5 28,-82.5 34,-82.5 34,-82.5 78,-82.5 78,-82.5 84,-82.5 90,-88.5 90,-94.5 90,-94.5 90,-107.5 90,-107.5 90,-113.5 84,-119.5 78,-119.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-97.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">output</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7f471e26ee90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%cell_to_module joke --display --execute node_to_execute\n",
    "def topic() -> str:\n",
    "    return \"Cowsay\"\n",
    "\n",
    "def joke_prompt(topic: str) -> str:\n",
    "    return f\"Knock, knock. Who's there? {topic}\"\n",
    "\n",
    "def reply(joke_prompt: str) -> str:\n",
    "    _, _, right = joke_prompt.partition(\"? \")\n",
    "    return f\"{right} who?\"\n",
    "\n",
    "def punchline(reply: str) -> str:\n",
    "    left, _, _ = reply.partition(\" \")\n",
    "    return f\"No, {left} MooOOooo\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08941f89",
   "metadata": {},
   "source": [
    "### 3.3 Inspecting results\n",
    "Ok, but how do you access results? With the node name! Magic 🧙"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "7ce116fc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DeprecationWarning: `--hide_results` is no longer required when using `--execute`. Now, results are hidden by default. Use `--show_results` if you want them automatically printed to cell output.\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"470pt\" height=\"222pt\"\n",
       " viewBox=\"0.00 0.00 470.00 222.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 218)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-218 466,-218 466,4 -4,4\"/>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>cluster__legend</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"8,-74 8,-206 104,-206 104,-74 8,-74\"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-190.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Legend</text>\n",
       "</g>\n",
       "<!-- joke_prompt -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>joke_prompt</title>\n",
       "<path fill=\"#ffc857\" stroke=\"black\" d=\"M232,-64C232,-64 137,-64 137,-64 131,-64 125,-58 125,-52 125,-52 125,-12 125,-12 125,-6 131,0 137,0 137,0 232,0 232,0 238,0 244,-6 244,-12 244,-12 244,-52 244,-52 244,-58 238,-64 232,-64\"/>\n",
       "<text text-anchor=\"start\" x=\"136\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">joke_prompt</text>\n",
       "<text text-anchor=\"start\" x=\"175\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- reply -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>reply</title>\n",
       "<path fill=\"#ffc857\" stroke=\"black\" d=\"M323,-64C323,-64 285,-64 285,-64 279,-64 273,-58 273,-52 273,-52 273,-12 273,-12 273,-6 279,0 285,0 285,0 323,0 323,0 329,0 335,-6 335,-12 335,-12 335,-52 335,-52 335,-58 329,-64 323,-64\"/>\n",
       "<text text-anchor=\"start\" x=\"284\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">reply</text>\n",
       "<text text-anchor=\"start\" x=\"294.5\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- joke_prompt&#45;&gt;reply -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>joke_prompt&#45;&gt;reply</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M244.25,-32C250.42,-32 256.59,-32 262.48,-32\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"262.74,-35.5 272.74,-32 262.74,-28.5 262.74,-35.5\"/>\n",
       "</g>\n",
       "<!-- topic -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>topic</title>\n",
       "<path fill=\"#ffc857\" stroke=\"black\" d=\"M75,-64C75,-64 37,-64 37,-64 31,-64 25,-58 25,-52 25,-52 25,-12 25,-12 25,-6 31,0 37,0 37,0 75,0 75,0 81,0 87,-6 87,-12 87,-12 87,-52 87,-52 87,-58 81,-64 75,-64\"/>\n",
       "<text text-anchor=\"start\" x=\"36\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">topic</text>\n",
       "<text text-anchor=\"start\" x=\"46.5\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- topic&#45;&gt;joke_prompt -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>topic&#45;&gt;joke_prompt</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M87.3,-32C95.71,-32 105.2,-32 114.85,-32\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"114.94,-35.5 124.94,-32 114.94,-28.5 114.94,-35.5\"/>\n",
       "</g>\n",
       "<!-- punchline -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>punchline</title>\n",
       "<path fill=\"#ffc857\" stroke=\"black\" d=\"M450,-64C450,-64 376,-64 376,-64 370,-64 364,-58 364,-52 364,-52 364,-12 364,-12 364,-6 370,0 376,0 376,0 450,0 450,0 456,0 462,-6 462,-12 462,-12 462,-52 462,-52 462,-58 456,-64 450,-64\"/>\n",
       "<text text-anchor=\"start\" x=\"375\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">punchline</text>\n",
       "<text text-anchor=\"start\" x=\"403.5\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- reply&#45;&gt;punchline -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>reply&#45;&gt;punchline</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M335.09,-32C340.93,-32 347.24,-32 353.64,-32\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"353.71,-35.5 363.71,-32 353.71,-28.5 353.71,-35.5\"/>\n",
       "</g>\n",
       "<!-- function -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>function</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M84,-174.5C84,-174.5 28,-174.5 28,-174.5 22,-174.5 16,-168.5 16,-162.5 16,-162.5 16,-149.5 16,-149.5 16,-143.5 22,-137.5 28,-137.5 28,-137.5 84,-137.5 84,-137.5 90,-137.5 96,-143.5 96,-149.5 96,-149.5 96,-162.5 96,-162.5 96,-168.5 90,-174.5 84,-174.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-152.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">function</text>\n",
       "</g>\n",
       "<!-- output -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>output</title>\n",
       "<path fill=\"#ffc857\" stroke=\"black\" d=\"M78,-119.5C78,-119.5 34,-119.5 34,-119.5 28,-119.5 22,-113.5 22,-107.5 22,-107.5 22,-94.5 22,-94.5 22,-88.5 28,-82.5 34,-82.5 34,-82.5 78,-82.5 78,-82.5 84,-82.5 90,-88.5 90,-94.5 90,-94.5 90,-107.5 90,-107.5 90,-113.5 84,-119.5 78,-119.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-97.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">output</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7f471f6c3190>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%cell_to_module joke --display --execute --hide_results\n",
    "def topic() -> str:\n",
    "    return \"Cowsay\"\n",
    "\n",
    "def joke_prompt(topic: str) -> str:\n",
    "    return f\"Knock, knock. Who's there? {topic}\"\n",
    "\n",
    "def reply(joke_prompt: str) -> str:\n",
    "    _, _, right = joke_prompt.partition(\"? \")\n",
    "    return f\"{right} who?\"\n",
    "\n",
    "def punchline(reply: str) -> str:\n",
    "    left, _, _ = reply.partition(\" \")\n",
    "    return f\"No, {left} MooOOooo\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "5715f0f7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'No, Cowsay MooOOooo'"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "punchline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9c24f8d",
   "metadata": {},
   "source": [
    "The results are assigned to variables matching their node names. This means you can quickly access them by typing their name. In most notebook environment, you get tab-completion for the and can view results in the variable inspector (especially useful for dataframes).\n",
    "\n",
    "What's the magic trick? 🐰\n",
    "\n",
    "When executing the cell, we are effectively:\n",
    "1. Loading it's content as a module\n",
    "2. Building a `Driver` with this module\n",
    "3. Executing the module with all nodes\n",
    "4. Assigning values from the results of `.execute()` to variables\n",
    "\n",
    "Consequently, functions defined in `%%cell_to_node` are replaced by their \"value\". You access functions directly through their module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "c6123ec7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'No, Foxey MooOOooo'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joke.punchline(\"Foxey\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6c6dc7a",
   "metadata": {},
   "source": [
    "### 3.4 Inputs & overrides\n",
    "In Hamilton, *inputs*  are values external to the dataflow and *overrides* are values to replace the output of a node (it effectively skips upstream operations). You can use `--inputs / -i` and `--outputs / -o` to pass dictionaries of values for execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "fd88cda5",
   "metadata": {},
   "outputs": [],
   "source": [
    "my_inputs = dict(topic=\"monday\")\n",
    "my_overrides = dict(punchline=\"Bingo bongo!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "1ab50a49",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"475pt\" height=\"322pt\"\n",
       " viewBox=\"0.00 0.00 474.50 322.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 318)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-318 470.5,-318 470.5,4 -4,4\"/>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>cluster__legend</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"8,-64 8,-306 104,-306 104,-64 8,-64\"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-290.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Legend</text>\n",
       "</g>\n",
       "<!-- reply -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>reply</title>\n",
       "<path fill=\"#ffc857\" stroke=\"black\" d=\"M327.5,-64C327.5,-64 289.5,-64 289.5,-64 283.5,-64 277.5,-58 277.5,-52 277.5,-52 277.5,-12 277.5,-12 277.5,-6 283.5,0 289.5,0 289.5,0 327.5,0 327.5,0 333.5,0 339.5,-6 339.5,-12 339.5,-12 339.5,-52 339.5,-52 339.5,-58 333.5,-64 327.5,-64\"/>\n",
       "<text text-anchor=\"start\" x=\"288.5\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">reply</text>\n",
       "<text text-anchor=\"start\" x=\"299\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- punchline -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>punchline</title>\n",
       "<polygon fill=\"#ffc857\" stroke=\"black\" points=\"466.5,-64 368.5,-64 368.5,0 466.5,0 466.5,-64\"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"380.5,-64 368.5,-52 \"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"368.5,-12 380.5,0 \"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"454.5,0 466.5,-12 \"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"466.5,-52 454.5,-64 \"/>\n",
       "<text text-anchor=\"start\" x=\"379.5\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">punchline</text>\n",
       "<text text-anchor=\"start\" x=\"408\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- reply&#45;&gt;punchline -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>reply&#45;&gt;punchline</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M339.59,-32C345.43,-32 351.74,-32 358.14,-32\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"358.21,-35.5 368.21,-32 358.21,-28.5 358.21,-35.5\"/>\n",
       "</g>\n",
       "<!-- joke_prompt -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>joke_prompt</title>\n",
       "<path fill=\"#ffc857\" stroke=\"black\" d=\"M236.5,-64C236.5,-64 141.5,-64 141.5,-64 135.5,-64 129.5,-58 129.5,-52 129.5,-52 129.5,-12 129.5,-12 129.5,-6 135.5,0 141.5,0 141.5,0 236.5,0 236.5,0 242.5,0 248.5,-6 248.5,-12 248.5,-12 248.5,-52 248.5,-52 248.5,-58 242.5,-64 236.5,-64\"/>\n",
       "<text text-anchor=\"start\" x=\"140.5\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">joke_prompt</text>\n",
       "<text text-anchor=\"start\" x=\"179.5\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- joke_prompt&#45;&gt;reply -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>joke_prompt&#45;&gt;reply</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M248.75,-32C254.92,-32 261.09,-32 266.98,-32\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"267.24,-35.5 277.24,-32 267.24,-28.5 267.24,-35.5\"/>\n",
       "</g>\n",
       "<!-- _joke_prompt_inputs -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>_joke_prompt_inputs</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"100.5,-54.5 11.5,-54.5 11.5,-9.5 100.5,-9.5 100.5,-54.5\"/>\n",
       "<text text-anchor=\"start\" x=\"27\" y=\"-27.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">topic</text>\n",
       "<text text-anchor=\"start\" x=\"67\" y=\"-27.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- _joke_prompt_inputs&#45;&gt;joke_prompt -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>_joke_prompt_inputs&#45;&gt;joke_prompt</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M100.62,-32C106.68,-32 113.02,-32 119.39,-32\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"119.41,-35.5 129.41,-32 119.41,-28.5 119.41,-35.5\"/>\n",
       "</g>\n",
       "<!-- input -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>input</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" stroke-dasharray=\"5,2\" points=\"85.5,-274.5 26.5,-274.5 26.5,-237.5 85.5,-237.5 85.5,-274.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-252.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">input</text>\n",
       "</g>\n",
       "<!-- function -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>function</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M84,-219.5C84,-219.5 28,-219.5 28,-219.5 22,-219.5 16,-213.5 16,-207.5 16,-207.5 16,-194.5 16,-194.5 16,-188.5 22,-182.5 28,-182.5 28,-182.5 84,-182.5 84,-182.5 90,-182.5 96,-188.5 96,-194.5 96,-194.5 96,-207.5 96,-207.5 96,-213.5 90,-219.5 84,-219.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-197.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">function</text>\n",
       "</g>\n",
       "<!-- output -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>output</title>\n",
       "<path fill=\"#ffc857\" stroke=\"black\" d=\"M78,-164.5C78,-164.5 34,-164.5 34,-164.5 28,-164.5 22,-158.5 22,-152.5 22,-152.5 22,-139.5 22,-139.5 22,-133.5 28,-127.5 34,-127.5 34,-127.5 78,-127.5 78,-127.5 84,-127.5 90,-133.5 90,-139.5 90,-139.5 90,-152.5 90,-152.5 90,-158.5 84,-164.5 78,-164.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-142.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">output</text>\n",
       "</g>\n",
       "<!-- override -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>override</title>\n",
       "<polygon fill=\"#b4d8e4\" stroke=\"black\" points=\"96,-109.5 16,-109.5 16,-72.5 96,-72.5 96,-109.5\"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"28,-109.5 16,-97.5 \"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"16,-84.5 28,-72.5 \"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"84,-72.5 96,-84.5 \"/>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"96,-97.5 84,-109.5 \"/>\n",
       "<text text-anchor=\"middle\" x=\"56\" y=\"-87.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">override</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7f471e26e3d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%cell_to_module joke --display --execute --inputs my_inputs --overrides my_overrides\n",
    "def joke_prompt(topic: str) -> str:\n",
    "    return f\"Knock, knock. Who's there? {topic}\"\n",
    "\n",
    "def reply(joke_prompt: str) -> str:\n",
    "    _, _, right = joke_prompt.partition(\"? \")\n",
    "    return f\"{right} who?\"\n",
    "\n",
    "def punchline(reply: str) -> str:\n",
    "    left, _, _ = reply.partition(\" \")\n",
    "    return f\"No, {left} MooOOooo\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "402003ce",
   "metadata": {},
   "source": [
    "### 3.5 Driver Adapters\n",
    "You can modify execution by passing adapters to `--builder / -b`. Adapters are flexible tools that can provide a variety of features. For instance, the next few cells uses `PrintLn()` to execution status after each node."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "2346e843",
   "metadata": {},
   "outputs": [],
   "source": [
    "from hamilton.lifecycle.default import PrintLn\n",
    "my_builder = driver.Builder().with_adapters(PrintLn())  # add the adapter"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed0a1e34",
   "metadata": {},
   "source": [
    "Notice in the printed statement how the overriden `punchline` node isn't executed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "0b3b0ba0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"345pt\" height=\"222pt\"\n",
       " viewBox=\"0.00 0.00 345.00 222.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 218)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-218 341,-218 341,4 -4,4\"/>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>cluster__legend</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"11.5,-74 11.5,-206 107.5,-206 107.5,-74 11.5,-74\"/>\n",
       "<text text-anchor=\"middle\" x=\"59.5\" y=\"-190.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Legend</text>\n",
       "</g>\n",
       "<!-- joke_prompt -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>joke_prompt</title>\n",
       "<path fill=\"#ffc857\" stroke=\"black\" d=\"M107,-64C107,-64 12,-64 12,-64 6,-64 0,-58 0,-52 0,-52 0,-12 0,-12 0,-6 6,0 12,0 12,0 107,0 107,0 113,0 119,-6 119,-12 119,-12 119,-52 119,-52 119,-58 113,-64 107,-64\"/>\n",
       "<text text-anchor=\"start\" x=\"11\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">joke_prompt</text>\n",
       "<text text-anchor=\"start\" x=\"50\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- reply -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>reply</title>\n",
       "<path fill=\"#ffc857\" stroke=\"black\" d=\"M198,-64C198,-64 160,-64 160,-64 154,-64 148,-58 148,-52 148,-52 148,-12 148,-12 148,-6 154,0 160,0 160,0 198,0 198,0 204,0 210,-6 210,-12 210,-12 210,-52 210,-52 210,-58 204,-64 198,-64\"/>\n",
       "<text text-anchor=\"start\" x=\"159\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">reply</text>\n",
       "<text text-anchor=\"start\" x=\"169.5\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- joke_prompt&#45;&gt;reply -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>joke_prompt&#45;&gt;reply</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M119.25,-32C125.42,-32 131.59,-32 137.48,-32\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"137.74,-35.5 147.74,-32 137.74,-28.5 137.74,-35.5\"/>\n",
       "</g>\n",
       "<!-- punchline -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>punchline</title>\n",
       "<path fill=\"#ffc857\" stroke=\"black\" d=\"M325,-64C325,-64 251,-64 251,-64 245,-64 239,-58 239,-52 239,-52 239,-12 239,-12 239,-6 245,0 251,0 251,0 325,0 325,0 331,0 337,-6 337,-12 337,-12 337,-52 337,-52 337,-58 331,-64 325,-64\"/>\n",
       "<text text-anchor=\"start\" x=\"250\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">punchline</text>\n",
       "<text text-anchor=\"start\" x=\"278.5\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- reply&#45;&gt;punchline -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>reply&#45;&gt;punchline</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M210.09,-32C215.93,-32 222.24,-32 228.64,-32\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"228.71,-35.5 238.71,-32 228.71,-28.5 228.71,-35.5\"/>\n",
       "</g>\n",
       "<!-- function -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>function</title>\n",
       "<path fill=\"#b4d8e4\" stroke=\"black\" d=\"M87.5,-174.5C87.5,-174.5 31.5,-174.5 31.5,-174.5 25.5,-174.5 19.5,-168.5 19.5,-162.5 19.5,-162.5 19.5,-149.5 19.5,-149.5 19.5,-143.5 25.5,-137.5 31.5,-137.5 31.5,-137.5 87.5,-137.5 87.5,-137.5 93.5,-137.5 99.5,-143.5 99.5,-149.5 99.5,-149.5 99.5,-162.5 99.5,-162.5 99.5,-168.5 93.5,-174.5 87.5,-174.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"59.5\" y=\"-152.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">function</text>\n",
       "</g>\n",
       "<!-- output -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>output</title>\n",
       "<path fill=\"#ffc857\" stroke=\"black\" d=\"M81.5,-119.5C81.5,-119.5 37.5,-119.5 37.5,-119.5 31.5,-119.5 25.5,-113.5 25.5,-107.5 25.5,-107.5 25.5,-94.5 25.5,-94.5 25.5,-88.5 31.5,-82.5 37.5,-82.5 37.5,-82.5 81.5,-82.5 81.5,-82.5 87.5,-82.5 93.5,-88.5 93.5,-94.5 93.5,-94.5 93.5,-107.5 93.5,-107.5 93.5,-113.5 87.5,-119.5 81.5,-119.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"59.5\" y=\"-97.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">output</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7f471e279210>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing node: joke_prompt.\n",
      "Finished debugging node: joke_prompt in 80.3μs. Status: Success.\n",
      "Executing node: reply.\n",
      "Finished debugging node: reply in 44.8μs. Status: Success.\n",
      "Executing node: punchline.\n",
      "Finished debugging node: punchline in 42.7μs. Status: Success.\n"
     ]
    }
   ],
   "source": [
    "%%cell_to_module joke --builder my_builder --display --execute \n",
    "def joke_prompt() -> str:\n",
    "    return f\"Knock, knock. Who's there? Cowsay\"\n",
    "\n",
    "def reply(joke_prompt: str) -> str:\n",
    "    _, _, right = joke_prompt.partition(\"? \")\n",
    "    return f\"{right} who?\"\n",
    "\n",
    "def punchline(reply: str) -> str:\n",
    "    left, _, _ = reply.partition(\" \")\n",
    "    return f\"No, {left} MooOOooo\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e07c0b4",
   "metadata": {},
   "source": [
    "There are ton of awesome adapters that can help you with your notebook experience. Here are a few notable mentions:\n",
    "\n",
    "1. `hamilton.lifecycle.default.PrintLn()` print execution status (this is similar to the `--show_results` flag).\n",
    "2. `hamilton.plugins.h_tqdm.ProgressBar()` add a progress bar for execution.\n",
    "3. `hamilton.lifecycle.default.PDBDebugger()` allows you to step into a node with a Python debugger, allowing you to execute code line by line.\n",
    "\n",
    "Note that all of these adapters work with Hamilton outside notebooks too!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80779b60",
   "metadata": {},
   "source": [
    "## 3.6 Caching\n",
    "The next builder include the `.with_cache()` clause. This will store results on disks. Adding the `--display_cache` flag will print a visualization of what results were retrieved from cache once execution is completed.\n",
    "\n",
    "To view it, execute the cell twice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "2a43046b",
   "metadata": {},
   "outputs": [],
   "source": [
    "builder_with_cache = driver.Builder().with_cache()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "0ec9d3bf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"352pt\" height=\"222pt\"\n",
       " viewBox=\"0.00 0.00 351.50 222.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 218)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-218 347.5,-218 347.5,4 -4,4\"/>\n",
       "<g id=\"clust1\" class=\"cluster\">\n",
       "<title>cluster__legend</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"8,-74 8,-206 124,-206 124,-74 8,-74\"/>\n",
       "<text text-anchor=\"middle\" x=\"66\" y=\"-190.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">Legend</text>\n",
       "</g>\n",
       "<!-- joke_prompt -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>joke_prompt</title>\n",
       "<path fill=\"#ffc857\" stroke=\"#f06449\" stroke-width=\"3\" d=\"M113.5,-64C113.5,-64 18.5,-64 18.5,-64 12.5,-64 6.5,-58 6.5,-52 6.5,-52 6.5,-12 6.5,-12 6.5,-6 12.5,0 18.5,0 18.5,0 113.5,0 113.5,0 119.5,0 125.5,-6 125.5,-12 125.5,-12 125.5,-52 125.5,-52 125.5,-58 119.5,-64 113.5,-64\"/>\n",
       "<text text-anchor=\"start\" x=\"17.5\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">joke_prompt</text>\n",
       "<text text-anchor=\"start\" x=\"56.5\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- reply -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>reply</title>\n",
       "<path fill=\"#ffc857\" stroke=\"#f06449\" stroke-width=\"3\" d=\"M204.5,-64C204.5,-64 166.5,-64 166.5,-64 160.5,-64 154.5,-58 154.5,-52 154.5,-52 154.5,-12 154.5,-12 154.5,-6 160.5,0 166.5,0 166.5,0 204.5,0 204.5,0 210.5,0 216.5,-6 216.5,-12 216.5,-12 216.5,-52 216.5,-52 216.5,-58 210.5,-64 204.5,-64\"/>\n",
       "<text text-anchor=\"start\" x=\"165.5\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">reply</text>\n",
       "<text text-anchor=\"start\" x=\"176\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- joke_prompt&#45;&gt;reply -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>joke_prompt&#45;&gt;reply</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M125.75,-32C131.92,-32 138.09,-32 143.98,-32\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"144.24,-35.5 154.24,-32 144.24,-28.5 144.24,-35.5\"/>\n",
       "</g>\n",
       "<!-- punchline -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>punchline</title>\n",
       "<path fill=\"#ffc857\" stroke=\"#f06449\" stroke-width=\"3\" d=\"M331.5,-64C331.5,-64 257.5,-64 257.5,-64 251.5,-64 245.5,-58 245.5,-52 245.5,-52 245.5,-12 245.5,-12 245.5,-6 251.5,0 257.5,0 257.5,0 331.5,0 331.5,0 337.5,0 343.5,-6 343.5,-12 343.5,-12 343.5,-52 343.5,-52 343.5,-58 337.5,-64 331.5,-64\"/>\n",
       "<text text-anchor=\"start\" x=\"256.5\" y=\"-42.8\" font-family=\"Helvetica,sans-Serif\" font-weight=\"bold\" font-size=\"14.00\">punchline</text>\n",
       "<text text-anchor=\"start\" x=\"285\" y=\"-14.8\" font-family=\"Helvetica,sans-Serif\" font-style=\"italic\" font-size=\"14.00\">str</text>\n",
       "</g>\n",
       "<!-- reply&#45;&gt;punchline -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>reply&#45;&gt;punchline</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M216.59,-32C222.43,-32 228.74,-32 235.14,-32\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"235.21,-35.5 245.21,-32 235.21,-28.5 235.21,-35.5\"/>\n",
       "</g>\n",
       "<!-- output -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>output</title>\n",
       "<path fill=\"#ffc857\" stroke=\"black\" d=\"M88,-174.5C88,-174.5 44,-174.5 44,-174.5 38,-174.5 32,-168.5 32,-162.5 32,-162.5 32,-149.5 32,-149.5 32,-143.5 38,-137.5 44,-137.5 44,-137.5 88,-137.5 88,-137.5 94,-137.5 100,-143.5 100,-149.5 100,-149.5 100,-162.5 100,-162.5 100,-168.5 94,-174.5 88,-174.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"66\" y=\"-152.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">output</text>\n",
       "</g>\n",
       "<!-- from cache -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>from cache</title>\n",
       "<path fill=\"#ffffff\" stroke=\"#f06449\" stroke-width=\"3\" d=\"M104,-119.5C104,-119.5 28,-119.5 28,-119.5 22,-119.5 16,-113.5 16,-107.5 16,-107.5 16,-94.5 16,-94.5 16,-88.5 22,-82.5 28,-82.5 28,-82.5 104,-82.5 104,-82.5 110,-82.5 116,-88.5 116,-94.5 116,-94.5 116,-107.5 116,-107.5 116,-113.5 110,-119.5 104,-119.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"66\" y=\"-97.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">from cache</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x7f47433eeb10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%cell_to_module joke --builder builder_with_cache --execute --display_cache\n",
    "def joke_prompt() -> str:\n",
    "    return f\"Knock, knock. Who's there? Cowsay\"\n",
    "\n",
    "def reply(joke_prompt: str) -> str:\n",
    "    _, _, right = joke_prompt.partition(\"? \")\n",
    "    return f\"{right} who?\"\n",
    "\n",
    "def punchline(reply: str) -> str:\n",
    "    left, _, _ = reply.partition(\" \")\n",
    "    return f\"No, {left} MooOOooo\""
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
